Papers with text encoding
Extracting Text Representations for Terms and Phrases in Technical Domains (2023.acl-industry)
Copied to clipboard
| Challenge: | Large pre-trained language models are extensively used in modern NLP systems. |
| Approach: | They propose an unsupervised approach to encoding using character-based models and pre-trained sentence encoders to reconstruct large pre-trained embedding matrices. |
| Outcome: | The proposed approach matches the quality of sentence encoders in technical domains and is 5 times smaller and up to 10 times faster on high-end GPUs. |
Padding Tone: A Mechanistic Analysis of Padding Tokens in T2I Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Text-to-image (T2I) diffusion models rely on encoded prompts to guide the image generation process. |
| Approach: | They conduct the first in-depth analysis of the role padding tokens play in T2I diffusion models by using two causal techniques to analyze how information is encoded in the representation of tokens across different components of the pipeline. |
| Outcome: | The proposed techniques reveal that padding tokens may affect the model’s output during text encoding, during the diffusion process, or be effectively ignored. |
Randomly Removing 50% of Dimensions in Text Embeddings has Minimal Impact on Retrieval and Classification Tasks (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on text embeddings focus less on how information is encoded. |
| Approach: | They find that truncating embedding dimensions causes an increase in performance when removed. |
| Outcome: | The proposed method improves performance across 6 state-of-the-art text encoders and 26 downstream tasks. |